Papers with Transformer baseline
Two-Headed Monster and Crossed Co-Attention Networks (2020.aacl-srw)
Copied to clipboard
| Challenge: | a new co-attentional neural structure is proposed for machine translation tasks . a higher-level and more abstract paradigm generalized from CCNs is proposed . |
| Approach: | They propose a paradigm that consists of two symmetric encoder modules and one decoder module connected with co-attention. |
| Outcome: | The proposed model outperforms the current Transformer model on translation tasks but the epoch time increases by circa 75%. |
Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing non-autoregressive neural machine translation models are slow to learn the dependency between output tokens. |
| Approach: | They propose to use fully non-autoregressive neural machine translation (NAT) to predict tokens with single forward of neural networks. |
| Outcome: | The proposed model achieves state-of-the-art results on three translation benchmarks with comparable performance to autoregressive and iterative NAT systems. |
To Clarify or not to Clarify: A Comparative Analysis of Clarification Classification with Fine-Tuning, Prompt Tuning, and Prompt Engineering (2024.naacl-srw)
Copied to clipboard
| Challenge: | Xu et al., 2019) show that pre-trained language model fine-tuning and prompt tuning are better than manual prompt engineering for clarification identification. |
| Approach: | They propose to use pre-trained language model fine-tuning, prompt tuning and manual prompt engineering to model clarification identification. |
| Outcome: | The proposed model outperforms pre-trained language model fine-tuning, prompt tuning and manual prompt engineering on the task of clarification identification. |
Mixed Multi-Head Self-Attention for Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Recent advances in neural machine translation have been made in the field of multi-head self-attention and there is no explicit mechanism to ensure that different attention heads capture different features. |
| Approach: | They propose a novel multi-head self-attention model which models not only global and local attention but also forward and backward attention in different attention heads. |
| Outcome: | The proposed model improves on WAT17 English-Japanese and IWSLT14 German-English translation tasks without increasing the number of parameters. |
Learning Source Phrase Representations for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to machine translation have been shown to be effective for long sentences . however, the attentional network can't capture long-distance dependencies . |
| Approach: | They propose a multi-head attention mechanism which generates phrase representations from token representations and incorporates them into the Transformer translation model to enhance its ability to capture long-distance relationships. |
| Outcome: | The proposed model can be computed in parallel and improves on the WMT 14 tasks. |
Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to token-level adaptive training only use static word frequency information without considering the source language. |
| Approach: | They propose a bilingual mutual information based adaptive objective that assigns weights to target tokens with higher BMI . they propose to use this approach to improve token-level adaptive training . |
| Outcome: | The proposed method improves token-level adaptive training on two languages. |
Hierarchical Transformers Are More Efficient Language Models (2022.findings-naacl)
Copied to clipboard
Piotr Nawrot, Szymon Tworkowski, Michał Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, Henryk Michalewski
| Challenge: | Transformers are impressive but inefficient and costly, which limits their applications and accessibility. |
| Approach: | They first use different ways to downsample and upsamplify activations in Transformers to make them hierarchical. |
| Outcome: | The proposed model outperforms Transformers on the ImageNet32 and enwik8 benchmarks. |
Recurrent Positional Embedding for Neural Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Existing translation systems that use positional embeddings only encode static order dependencies based on discrete numerical information, which may hinder the improvement of translation capacity. |
| Approach: | They propose a recurrent positional embedding approach based on word vectors that are learned by a neural network and integrated into existing multi-head self-attention models. |
| Outcome: | The proposed approach improves translation performance over the state-of-the-art Transformer baseline in English-to-German and NIST Chinese-to English translation tasks. |
A Comparison between Pre-training and Large-scale Back-translation for Neural Machine Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | BERT is a promising technique to improve NMT, but how it outperforms standard NMT is understudied. |
| Approach: | We compare MT engines trained with pre-trained BERT and back-translation with incrementally larger amounts of data. |
| Outcome: | The proposed technique outperforms standard NMT models on morphology and syntax. |
Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to improve neural machine translation use token-level adaptive training . however, standard models make predictions on condition of previous contexts . |
| Approach: | They propose a target-context-aware metric which can be supplemented by statistical metrics . they propose an adaptive training approach based on token- and sentence-level CBMI . |
| Outcome: | The proposed model outperforms the Transformer baseline and other similar approaches on English-German and Chinese-English tasks. |
Scheduled Sampling Based on Decoding Steps for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Neural machine translation models are trained to maximize the likelihood of next token given previous golden tokens as inputs, but at the inference stage, golden token is unavailable. |
| Approach: | They propose to use scheduled sampling to replace ground-truth tokens with predicted tokens to bridge the gap between training and inference. |
| Outcome: | The proposed methods outperform the Transformer baseline and vanilla scheduled sampling on three large-scale WMT tasks. |
Prevent the Language Model from being Overconfident in Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation models are based on partial translation and a language model that predicts the next token based only on partial. |
| Approach: | They propose a Margin-based Token-level Objective and a Sentence-level Goal to maximize the Margin . they propose to model the next token based on partial translation . |
| Outcome: | The proposed approach improves translation adequacy and fluency on English-to-German, Chinese-to English and French translation tasks. |
Token Drop mechanism for Neural Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Neural machine translation models are vulnerable to unfamiliar inputs. |
| Approach: | They propose to drop tokens of the input sentences to improve generalization and avoid overfitting for the NMT model. |
| Outcome: | The proposed approach improves on Chinese-English and English-Romanian benchmarks and achieves significant performance improvements over baselines. |
SMRT Chatbots: Improving Non-Task-Oriented Dialog with Simulated Multiple Reference Training (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Simulated Multiple Reference Training (SMRT) improves non-task-oriented dialog models by reducing the need for related-domain dialog data. |
| Approach: | They apply Simulated Multiple Reference Training (SMRT) to chatbots to overcome sparse dialog data. |
| Outcome: | The proposed model outperforms pretraining on human evaluation quality and lexical diversity without requiring related-domain dialog data. |
Convolutional Self-Attention Networks (N19-1)
Copied to clipboard
| Challenge: | Existing models of self-attention networks lack the ability to capture dependencies regardless of distance and can be enhanced with multi-head attention. |
| Approach: | They propose a convolutional self-attention network which can be enhanced by multi-head attention by allowing the model to attend to information from different representation subspaces. |
| Outcome: | The proposed model outperforms existing models on improving locality of SANs on different language pairs and model settings. |
Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to improve online inference efficiency of the Transformer for instantaneous Grammatical Error Correction (GEC) are sequenceto-sequence (seq2sequ) and sequenceto sequence (saq2eq) |
| Approach: | They propose a novel approach to improve the online inference efficiency of the Transformer model for instantaneous Grammatical Error Correction (GEC) it aggressively decodes as many tokens as possible in parallel instead of always decoding only one token in each step to improve computational parallelism. |
| Outcome: | The proposed approach can achieve state-of-the-art results in English and Chinese benchmarks with 10x speedup over the Transformer-big model. |
Categorizing Semantic Representations for Neural Machine Translation (2022.coling-1)
Copied to clipboard
| Challenge: | Modern neural machine translation models suffer limitation in compositional generalization, resulting in weakened translation performance on unseen compounds. |
| Approach: | They propose to introduce categorization to the contextualized representations to improve generalization by reducing sparsity and overfitting. |
| Outcome: | The proposed method reduces compositional generalization error rates by 24% on a dedicated MT dataset. |
Selective Knowledge Distillation for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation models achieve state-of-the-art performance on many translation benchmarks. |
| Approach: | They propose a protocol that analyzes different impacts of samples by comparing various samples’ partitions. |
| Outcome: | The proposed methods yield up to +1.28 and +0.89 BLEU points improvements over the Transformer baseline, respectively. |
Linear Recency Bias During Training Improves Transformers’ Fit to Reading Times (2025.coling-main)
Copied to clipboard
| Challenge: | Recent research has shown a strong fit between surprisal values from Transformers and reading times. |
| Approach: | They evaluate a Transformer model that uses a recency bias added to attention scores to improve the fit to human reading times. |
| Outcome: | The proposed model improves on a Transformer that includes a recency bias added to attention scores. |
Guiding Neural Machine Translation with Semantic Kernels (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Empirical studies show that our approach gains approximately an improvement of 1 BLEU score on most benchmarks over the Transformer baseline. |
| Approach: | They propose to extract several semantic kernels from a source sentence to capture global semantic information. |
| Outcome: | Empirical results show that the proposed approach improves 1 BLEU score on benchmarks . it is also 1.7 times faster than previous works on average at inference time . |
Regularized Context Gates on Transformer for Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Context gates are effective to control the contributions from the source and target contexts in the recurrent neural network (RNN) based neural machine translation. |
| Approach: | They propose a method to identify source and target contexts and introduce a gate mechanism to control the contributions from source and targets in the advanced Transformer architecture. |
| Outcome: | The proposed model achieves an averaged gain of 1.0 BLEU score over a strong transformer baseline. |